Ph . D .

Anomaly detection is a challenging problem that has been researched within a variety of application domains such as image processing, fault isolation, fraud detection, and network intrusion detection. In network intrusion detection, anomaly based techniques are particularly attractive because of their ability to identify previously unknown attacks without the need to be programmed with the specific signatures of every possible attack. There is a significant body of work in anomaly based intrusion detection applying statistical analysis, data-mining, and machine learning disciplines. However despite more than two decades of active research, there is a striking lack of anomaly based systems in commercial use today. Many of the currently proposed anomaly based systems do not adequately address a series of challenges making them unsuitable for operational deployment. In existing approaches, every step of the anomaly detection process requires expert manual intervention. This dependence makes developing practical systems extremely challenging. In this thesis, we propose to integrate the strengths of machine learning and quality-of-service mitigation techniques for network anomaly detection, and build an operationally practical framework for anomaly based network intrusion detection. We will devise methods for self-adaptive, self-tuning, selfoptimizing, and automatically responsive network anomaly detection. In specific, we will propose methods for adaptive input normalization that will adjust scaling parameters online based on evolving values in observed traffic patterns. We will propose algorithms for dynamic threshold adaptation that will identify both small aggregate anomalies and single point anomalies while adaptively adjusting to account for concept drift. We will propose a model for dictating optimal performance in an anomaly detection system and propose responsive reinforcement learning algorithms for automated tuning and optimization. We will develop algorithms for dynamic anomaly classification into an evolving attack taxonomy and propose a model for confidence forwarding to feed an automated response engine. We will develop a fair bandwidth sharing and delay differentiation mechanism for scalable automated response to a variety of network attacks that will insulate network resources from malicious traffic while minimizing collateral damage. We will model a prototype network anomaly detection system that integrates the proposed and developed techniques. Extensive experiments will be conducted by using the 1999 Knowledge Discovery and Data-mining Cup datasets, but also we will create a new dataset based on a combination of live network traces and controlled simulated data injects (TD-SIM). As we formulate these ideas, we aim to advance the understanding of the practical capabilities and limitations of anomaly based intrusion detection. By successfully completing this research, we will advance the field by demonstrating an effective and efficient framework that can be applied to real world operational networks.